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ABSTRACT. The maximally clustered permutations are characterized by avoiding the classical permuta- 
tion patterns {3421, 4312, 4321}. This class contains the freely braided permutations and the fully com- 
mutative permutations. In this work, we show that the generating functions for certain fully commutative 
pattern classes can be transformed to give generating functions for the corresponding freely braided and 
maximally clustered pattern classes. Moreover, this transformation of generating functions is rational. As a 
result, we obtain enumerative formulas for the pattern classes mentioned above as well as the corresponding 
hexagon-avoiding pattern classes where the hexagon-avoiding permutations are characterized by avoiding 
{46718235, 46781235, 56718234, 56781234}. 



1. Introduction 

The maximally clustered permutations introduced in [Los07] are a generalization of the freely braided 
permutations developed in [GL02] and [GL04], and these in turn include the fully commutative permu- 
tations studied in [Ste96] as a subset. In [Jon07], an explicit formula was obtained for the Kazhdan- 
Lusztig polynomials of maximally-clustered hexagon-avoiding permutations, generalizing an earlier re- 
sult of [BW01] that identified the 321-hexagon avoiding permutations. The enumeration of the 321- 
hexagon avoiding permutations was first given by [SW04] who showed that these elements satisfy a 
linear constant-coefficient recurrence with 7 terms. 

Theorem 1.1. [SW04] The number c n of 321 -hexagon-avoiding permutations in S n satisfies the recur- 
rence 



C n +1 = 6c n - llc n _i + 9c n _ 2 



4c„ 



4c n _ 4 + c n _ 



for all n > 8 with initial conditions given in Figure 1. 



Theorem 1.1 was extended in [MS03] and also proved using an enumeration scheme as described 
in [Vat08]. Figure 1 shows the number b n and m n of freely-braided hexagon-avoiding and maximally- 
clustered hexagon- avoiding permutations respectively, in S n for n < 15. It can be shown that each of 
these pattern classes have a rational generating function by Proposition 13 and Corollary 10 of [ALR05]. 
In this paper, we develop a method for determining the generating function precisely. 
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FIGURE 1 . Enumeration of hexagon-avoiding classes 

One feature of our main results, Theorem 2.4 and Theorem 3.6 below, is that they provide many ex- 
amples of classes characterized by permutation pattern avoidance that have rational generating functions. 
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We also apply Theorem 2.4 to enumerate the maximally-clustered permutations introduced in [Los07]. 
The freely braided permutations have previously been enumerated in [Man04] and we recover this result 
from Theorem 2.4 as well. 

Section 1 describes the combinatorial model used in our enumeration. In Section 2 we give a result 
that enables us to find enumerative formulas for the pattern classes mentioned above in a unified way. 
In Section 3 we show how the methods of proof from Section 2 can be applied in the fully commutative 
case to recover the main result of [SW04] as well as some new generating functions. 

1.1. Background. We view the symmetric group S n as the Coxeter group of type A with generating set 
S = {si, . . . , s n -\} and relations of the form (sjSj-ti) 3 = 1 together with (siSj) 2 = 1 for \i — j\ > 2 
and s 2 = 1. We also refer to elements in the symmetric group by the 1-line notation w = \w\W2 ■ ■ ■ w n ] 
where w is the bijection mapping i to Then the generators Sj are the adjacent transpositions inter- 
changing the entries i and i + 1 in the 1-line notation. Suppose w = [wi . . . w n ] and p = [pi ■ ■ ■ Pk] is 
another permutation in for k < n. We say w contains the permutation pattern p or w contains p as a 
1-line pattern whenever there exists a subsequence 1 < i\ < < ■ ■ ■ < ik < n such that 

w i a < w i b if and onlv if Pa < Pb 

for all 1 < a < b < k. We call 12, ■ ■ ■ ,ik) the pattern instance. For example, [53241] contains the 
pattern [321] in several ways, including the underlined subsequence. If w does not contain the pattern 
p, we say that w avoids p. A pattern class is a set of permutations characterized by avoiding a set of 
permutation patterns. For example, the maximally clustered permutations are characterized by avoiding 
the permutation patterns 

(1.1) [3421], [4312], and [4321] 

by [Los07, Proposition 3.7], while the freely braided permutations are characterized by avoiding 

(1.2) [4231], [3421], [4312], and [4321] 

as permutation patterns by [GL02, Proposition 5.1.1]. 

Recall that the products of generators from S with a minimal number of factors are called reduced 
expressions, and l(w) is the length of such an expression for w 6 S n . Given w E S n , we represent 
reduced expressions for w in sans serif font, say w = W1W2 • • • w p where each G S. We call any 
expression of the form SiSi±\Si a short-braid. There is a well-known theorem of Matsumoto [Mat64] 
and Tits [Tit69], which states that any reduced expression for w can be transformed into any other by 
applying a sequence of relations of the form (s-iSi±i) 3 = 1 together with (siSj) 2 = 1 for \i — j\ > 1. The 
theorem implies that the set of all generators appearing in any reduced expression for w is well-defined. 
We call this set of generators the support of w and denote it by supp(w). We say that the permutation 
w is connected if the subscripts of the generators appearing in supp(w) form a nonempty interval in 
{l,2,...,n-l}. 

As in [Ste96], we define an equivalence relation on the set of reduced expressions for a permutation 
by saying that two reduced expressions are in the same commutativity class if one can be obtained from 
the other by a sequence of commuting moves of the form s-iSj 1— > SjSi where \i — j\ > 2. If the reduced 
expressions for a permutation w form a single commutativity class, then we say w is fully commutative. 

1.2. Heaps. If w = wi • • • is a reduced expression, then as in [Ste96] we define a partial ordering on 
the indices {1, • • • , k} by the transitive closure of the relation i -< j if i < j and Wj does not commute 
with Wj. We label each element i of the poset by the corresponding generator Wj. It follows from 
the definition that if w and w' are two reduced expressions for a permutation w that are in the same 
commutativity class, then the labeled posets of w and w' are isomorphic. This isomorphism class of 
labeled posets is called the heap of w, where w is a reduced expression representative for a commutativity 
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class of w. In particular, if w is fully commutative then it has a single commutativity class, and so there 
is a unique heap of w. 

As in [BW01], we will represent a heap as a set of lattice points embedded in N 2 . To do this, we 
assign coordinates (x,y) G N 2 to each entry of the labeled Hasse diagram for the heap of w in such a 
way that: 

(1) An entry represented by (x, y) is labeled s- L in the heap if and only if x = i, and 

(2) If an entry represented by (x, y) is greater than an entry represented by (V, y') in the heap, then 
V > y'- 

Since the Coxeter graph of type A is a path, it follows from the definition that (x, y) covers (x', y') in the 
heap if and only if x = x' ± 1, y > y', and there are no entries (x", y") such that x" G {x, x'} and y' < 
y" < y. Hence, we can completely reconstruct the edges of the Hasse diagram and the corresponding 
heap poset from a lattice point representation. This representation will enable us to make arguments "by 
picture" that would otherwise be difficult to formulate. Although there are many coordinate assignments 
for any particular heap, the x coordinates of each entry are fixed for all of them, and the coordinate 
assignments of any two entries only differs in the amount of vertical space between them. 

Example 1.2. One lattice point representation of the heap of w = S2S3S1S2S4 is shown below, together 
with the labeled Hasse diagram for the unique heap poset of w. 

si s 2 S3 sa 

Suppose x and y are a pair of entries in the heap of w that correspond to the same generator Sj, so 
that they lie in the same column i of the heap. Assume that x and y are a minimal pair in the sense that 
there is no other entry between them in column i. Then, for w to be reduced, there must exist at least one 
non-commuting generator between x and y, and if w is short-braid avoiding, there must actually be two 
non-commuting labeled heap entries that lie strictly between x and y in the heap. We call these two non- 
commuting labeled heap entries a resolution of the pair x, y. If the generators lie in distinct columns, 
we call the resolution a distinct resolution. The Lateral Convexity Lemma of [BW01] characterizes 
fully commutative permutations w as those for which every minimal pair in the heap of w has a distinct 
resolution. 

Definition 1.3. If Si G supp(w), we say that Si supports w. We also say that column i supports w. 
If w is fully commutative and connected then supp(w) = {sj, St+i> • • • > s j} f° r some i, j, and since 
every minimal pair of entries in the heap of w has a distinct resolution we must have exactly one entry in 
columns i and j of the heap. In this situation, we call columns i + l,i + 2, . . . ,j — 1 the internal columns 
of the heap of w and we call columns i and j the extremal columns of the heap of w. 

We now describe a notion of containment for heaps. Recall from [BJ07] that an orientation preserving 
Coxeter embedding f : {si, . . . , S£_i} — > {s\, s n _i} is an injective map of Coxeter generators such 
that for each m G {2, 3}, we have 

{s lSj ) m = 1 if and only if (f( Sl )f( Sj )) m = 1 

and the subscript of /(sj) is less than the subscript of f(sj) whenever i < j. We view this as a map of 
permutations, which we also denote / : S/. — > S n , by extending it to a word homomorphism which can 
then be applied to any reduced expression in 

Recall that a subposet Q of P is called convex if y G Q whenever x < y < z in P and x, z G Q. 
Suppose that w and h are permutations. We say that w heap-contains h if there exist commutativity 
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classes represented by w and h, together with an orientation preserving Coxeter embedding / such that 
the heap of /(h) is contained as a convex labeled subposet of the heap of w. If w does not heap-contain 
h, we say that w heap-avoids h. To illustrate, w = S2S3S1S2S4 from Example 1.2 heap-contains S1S2S3 
under the Coxeter embedding that sends Sj 1— > Sj+i, but w heap-avoids si^si- 

In type A, the heap construction can be combined with another combinatorial model for permutations 
in which the entries from the 1-line notation are represented by strings. The points at which two strings 
cross can be viewed as adjacent transpositions of the 1-line notation. Hence, we may overlay strings on 
top of a heap diagram to recover the 1-line notation for the permutation, by drawing the strings from 
bottom to top so that they cross at each entry in the heap where they meet and bounce at each lattice 
point not in the heap. Conversely, each permutation string diagram corresponds with a heap by taking all 
of the points where the strings cross as the entries of the heap. 

For example, we can overlay strings on the two heaps of [3214]. Note that the labels in the picture 
below refer to the strings, not the generators. 



3 2 1 4 3 2 1 4 




1 2 3 4 1 2 3 4 



For a more leisurely introduction to heaps and string diagrams, as well as generalizations to Coxeter 
types B and D, see [BJ07]. Cartier and Foata [CF69] were among the first to study heaps of dimers, 
which were generalized to other settings by Viennot [Vie89]. Stembridge has studied enumerative aspects 
of heaps [Ste96, Ste98] in the context of fully commutative elements. Green has also considered heaps 
of pieces with applications to Coxeter groups in [Gre03, Gre04a, Gre04b]. 

1.3. Maximally clustered elements. In [Los07], Losonczy introduced the maximally clustered ele- 
ments of simply laced Coxeter groups. 

Definition 1.4. [Los07] A braid cluster is an expression of the form 

Si 1 Si 2 . . . Si k Si k+1 Si k . . . s^s^ 

where each Si p for 1 < p < k has a unique Si q with p < q < k + 1 such that \i p — i q \ = 1. 

Let w be a permutation and let N(w) denote the number of [321] pattern instances in w. We say w is 
maximally clustered if there is a reduced expression for w of the form 

a ciaic 2 a 2 ■ . . c M a,M 

where each Oi is a reduced expression, each a is a braid cluster with length 2nj + l and N(w) = n «- 
Such an expression is called contracted. In particular, w is freely braided if there is a reduced expression 
for w with N(w) disjoint short-braids. 

This is not the original definition for the maximally clustered elements; however it is equivalent. The 
remarks in Section 5 of [GL02] show that the number of [321] pattern instances in w equals the number 
of contractible triples of roots in the inversion set of w. Corollary 4.11(h) and Corollary 4.13 of [Los07] 
prove that w is a contracted reduced expression for a maximally clustered element if and only if it has 
the form given in Definition 1.4. Moreover, it follows from the proof of [Los07, Corollary 4.11(ii)] that 
the a-i in Definition 1.4 are fully commutative. 

Recall that [BJS93] showed that w is fully commutative whenever N(w) = 0. In this work we will 
frequently use the fact that any braid cluster has the canonical form of Lemma 1.5. 



THE ENUMERATION OF MAXIMALLY CLUSTERED PERMUTATIONS 



5 



Lemma 1.5. Suppose x = s^s^ . . . Si k Si k+1 Si k . . . Sj 2 s u is a braid cluster of length 2k + 1 in type A. 
Then, x = s m+ is m+2 ■ ■ ■ s m +kSm+k+is m+ k ■ ■ ■ s m +2S m +ifor some m. 

Proof. Since Si k+1 is a transposition and conjugation preserves cycle type, x is a transposition and we 
may write x = (m + 1 m + k' + 2) in cycle notation for some k' ,m > 0. This transposition is given 
by the expression x = s m+ is m +2 ■ ■ ■ s m +k'Sm+k'+iSm+k' ■ ■ ■ s m +2S m +i, which is a reduced expression 
for x by [Los07, Lemma 4.3]. Since the length of reduced expressions for x is an invariant of x, k' = k 
and the result follows. □ 

Recall the following structural lemma about contracted reduced expressions. 

Lemma 1.6. [Jon07, Lemma 2.3] Let w be a contracted reduced expression for a maximally clustered 
permutation, so w has the form 

a ciai . . . c M a,M 

where each Cj is a braid cluster, and the a,- are short-braid avoiding. Then, any generator s, that appears 
in any of the braid clusters Cj does not appear anywhere else in w. 

Lemma 1.7. Let w be a contracted reduced expression for a maximally clustered permutation. If the 
generator Sj supports a braid cluster in w then Sj supports a braid cluster in every contracted reduced 
expression for w. 

Proof. By [Los07, Lemma 4.11(i)], the contracted reduced expressions form a complete set of represen- 
tatives for the commutativity classes of w. Suppose the generator s, supports a braid cluster in w in the 
sense of Definition 1.3 but there exists a contracted reduced expression w' = clqCicli . . . cmo-m for w 
in which Sj does not support a braid cluster. By the theorem of Matsumoto [Mat64] and Tits [Tit69], it 
suffices to consider a pair of heaps represented by w and w' that are related by a single short-braid move. 

Observe that each short-braid move on w' can change the length of at most one braid cluster. This 
is clear if the short-braid move involves two entries from a single braid cluster, or one entry from some 
braid cluster c; together with some entry of aj for j € {i — 1, i}. In the case that some cjj is the identity, 
note that there are no short-braid moves involving an entry from q and an entry from q+i by Lemma 1.6. 

Therefore, by the equation N(w) = J2iLi n i from Definition 1.4, we actually have that no short- 
braid move on w' can change the length of any of the braid clusters because N(w) does not depend 
on the reduced expression for w. Thus we have shown that the length of each braid cluster remains the 
same over all contracted reduced expressions for w. Finally, since no single short-braid move can change 
the support of a braid-cluster without changing its length, we have shown that the support of each braid 
cluster also remains the same over all contracted reduced expressions for w. □ 

Putting these lemmas together, we can show that there is a canonical heap associated to any maximally 
clustered permutation. See Example 1.10 for an illustration. 

Definition 1.8. Let £ be a commutativity class for a permutation w. Suppose that the set of columns 
{1, . . . , n} of the heap of £ can be partitioned into intervals 

C = [l,pi - l],Bi = [pi,gi],Ci = [qi + l,p 2 - 1], • • • ,B k = [pk,qk],C k = [q k + l,n] 

satisfying: 

(1) For each i, the heap of £ restricted to B{ is a braid cluster in the canonical form of Lemma 1.5. 

(2) For each i, every minimal pair of entries in any column from Q has a distinct resolution, possibly 
using entries from columns qi or pj+i. 

Then, we say that the heap of £ has a braid cluster column decomposition given by Co, B\ , C\ , . . . B^, Cfc. 
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Proposition 1.9. Every maximally clustered permutation w has a unique commutativity class £ such that 
the heap of <L has a braid cluster column decomposition. Conversely, every heap having a braid cluster 
column decomposition corresponds to a maximally clustered permutation. 

Proof. Let wbea maximally clustered permutation. By [Los07, Lemma 4.1 l(i)], the contracted reduced 
expressions form a complete set of representatives for the commutativity classes of w. By Lemma 1.7, 
each generator either supports a unique braid cluster in some contracted reduced expression, in which 
case the generator must also support the braid cluster in every contracted reduced expression, or the 
generator is not used in any braid cluster. Hence, the set of braid clusters appealing in any contracted 
reduced expression for w is independent of the choice of reduced expression. Therefore, specifying a 
commutativity class of w depends only on a choice of commutativity class for each braid cluster. We 
choose each braid cluster to have the form given in Lemma 1.5 and declare this to be our canonical 
commutativity class £ for w. 

Consider the heap associated to (E. We can partition the support of w into segments B\ , B 2 , . . . , Bk 
corresponding to the generators that support braid clusters. The complementary columns {1, . . . , n} \ 
Ui=i Bi form intervals that we denote Co, C\, . . . , C}.. 

By definition, this column partition satisfies property (1) of Definition 1.8. Any minimal pair of entries 
from some Ci without a distinct resolution would either form a short-braid or form an extension of an 
existing braid-cluster, contradicting the definition of the column partition we have given. Hence, the 
column partition satisfies property (2) of Definition 1.8 as well. 

To prove the converse statement, fix a heap with a column partition satisfying properties (1) and (2) 
and let w be a corresponding expression. We show that w is reduced and maximally clustered. 

If w is not reduced then there exists a sequence of braid moves bringing w to an expression whose 
heap has two entries in the same column with no entries between them. But the available braid moves 
in any expression related to w only involve entries from some braid cluster supported on columns Bi 
because the entries in the complimentary columns Ui=o ^» au nave distinct resolutions by property (2) 
and this remains true after performing arbitrarily many braid moves in U i=1 Bi- Since the braid clusters 
supported on each Bi are reduced, we never have any opportunity for cancellation. Hence, w is reduced. 

Moreover, we can read a contracted reduced expression for w from the given heap. We begin by 
linearly ordering the braid clusters B\ , . . . , B^ in a way which is compatible with the order in which 
they appear in the heap poset. First read all of the entries in columns C = Ui=o ^* tnat ^ e Deiow the 
entries of B\ in the heap poset. Then for each i < k we sequentially read the entries of B, L followed 
by the entries from columns C appearing between Bi and in the heap poset. Finally, we read the 
entries from columns C that appear above B& in the heap poset. 

Since the braid clusters have the canonical form of Lemma 1.5, we can verify that each braid cluster 
Bi supported on rij + 1 columns has n,i [321] -instances using a string diagram overlaid on the heap, 
as illustrated in Figure 2. By Lemma 1.6 each braid cluster Bi takes exactly one string lying next to a 
column in Ci and crosses it with exactly one string lying next to a column in Cj+i. Since each minimal 
pair of entries from any column of Ci has a distinct resolution by property (2), there are no other [321]- 
instances in w. Therefore, the number of [321] -instances in w is exactly X^to n *' an d eacn braid cluster 
Bi has length 2n-i + 1. Hence, w is maximally clustered by Definition 1.4. □ 

Example 1.10. Suppose w is given by the contracted reduced expression 
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Figure 2. Strings on a braid cluster 



Then the heap of w is drawn below, together with its braid cluster column decomposition. The braid 
clusters are shown in grey. 



[Bi 



[<5i ] [B 2 1 [<5 3 ] 



2. Enumeration 

Let S p = Un>i &n denote the permutations characterized by avoiding a set of 1-line patterns P. 
The most important pattern classes for this work are the maximally clustered permutations and the freely 
braided permutations, characterized by avoiding the patterns from (1.1) and (1.2), respectively. Given 
a finite set H of permutations, let S P (H) be the subset of S p consisting of those permutations that 
heap-avoid the patterns in H. 

In [Jon07, Section 4] we described how to find a set of 1-line patterns Q such that S p ({h}) = S®, 
when possible. Example 11.1 of [BJ07] shows that some heap patterns h have no such set Q. 

Let r(h) denote the rank of the symmetric group containing h. Define U p {h) to be the set of all 
elements in S p ^ that heap-contain h. This is a finite set because the rank is fixed. 

Definition 2.1. [Jon07, Definition 4.2] Let p G S . Then, we say that p is an ideal pattern in S p if for 
every q € S p ^ +1 containing p as a 1-line pattern, we have that q heap-contains p. 

For example, [Ten06, Theorem 3.8] implies that p is an ideal pattern in 5® if p avoids [2143]. Def- 
inition 2.1 describes a finite test that extends to permutations of all ranks according to the following 
result. 
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Theorem 2.2. [Jon07, Theorem 4.4] Suppose S P (H) is the subset of permutations characterized by 
avoiding a finite set P of 1 -line patterns and heap-avoiding a finite set H of permutations. If each of 
the elements in P' = (J/ieH U P (h) is an ideal pattern, then S P (H) = S p ^ p , so is characterized by 
avoiding the permutations in P{JP' as 1-line patterns. 

We can apply Theorem 2.2 to study certain classical permutation pattern classes using heap- avoidance. 
For example, it is straightforward to verify that 

S {[321]} ({s 5 S 6 S 7 S 3 S A S 5 S eS2 S 3 S 4 S 5Sl S 2S3 }) = 5{[321],[46718235],[46781235],[56718234],[56781234]} 

using Theorem 2.2. Similar statements [Jon07, Corollary 4.5] hold for the freely braided and maximally 
clustered permutations. The permutations that heap-avoid 

[46718235] = S5S6S7S3S4S5S6S2S3S4S5S1S2S3 
are called hexagon-avoiding after [BW01]. 

Let H be a finite set of connected fully commutative permutations each of whose heaps contains at 
least two entries in each internal column in the sense of Definition 1.3 or let H = 0. Suppose F n 
is the subset of the fully commutative permutations on n generators that are characterized by heap- 
avoiding the set of patterns from H, so F n = S^1^(H) C Sjf+i . We define \Fq\ to be 1, corre- 
sponding to the empty heap. Our main result in this section is that we can transform the generating 
function F(x) = Yl n >o \^ n \ x ™ to ODta i n generating functions for the corresponding freely braided and 
maximally clustered pattern classes. Moreover, the transformation is a rational function of F(x), so it 
preserves this important property of the generating function. 

We first consider the number of permutations in F n that have one or both of the extremal generators 
{si, s n } present in the heap. Recall that every minimal pair of entries in a fully commutative heap must 
have a distinct resolution, so in particular any fully commutative heap has at most a single entry in its 
leftmost and rightmost columns. Let L n be the permutations from F n that have s n present in their heap, 
and R n be the set of permutations from F n having s\ present in the heap. By the bijection that reverses 
the ordering of the subscripts of the generators, we find that \L n \ = \R n \. Let M n denote the set of 
permutations in F n with both s\ and s n present in the heap. Then, we have the following enumerative 
result. 

Lemma 2.3. Let H be a finite set of fully commutative permutations or let H = 0. Suppose 

F(x)=J2\S l n3(H)\x n - 

n>0 

Then, 

L(x) = \L n \ x n = F(x) - xF(x) - 1 

n>0 

and 

M(x) = \M n \ x n = F(x) - 2xF(x) + x 2 F(x) - 1. 

n>0 

Proof. This follows using inclusion-exclusion together with the observation that \Fq\ = 1 and = 2 
so [Lo| = I Mo I = I Mi I = 0, which is required by definition. □ 

We are now in a position to prove our main enumerative theorem. 

Theorem 2.4. Let H be a finite set of connected fully commutative permutations each of whose heaps 
contains at least two entries in each internal column or let H = 0. Suppose 

F(x)=J2\S l n3(H)\x n , 

n>0 
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L(x) = F(x) - xF(x) - 1, and M(x) = F(x) - 2xF(x) + x 2 F(x) - 1. 

Then, we have 

L{x) 2 



n>0 1 M ^ 



n>0 



{ [34 2 l],[4312],[43 2 l]} ( ^ )|a; „ = F(a;) + 



1 - x - M(x) ' 



Proof. Suppose w is an element of 5,^^ 21 ''' 4312 ''' 4321 ^(i7). By Proposition 1.9, we may choose a com- 
mutativity class (£ of w so that the heap of (£ has a specific form where each column either supports a 
braid cluster, in which case there are no other generators in that column, or else the column is not used 
in any braid cluster. 

Let k be the number of braid clusters in w. We partition the columns {1, . . . , n} of the heap of £ 
into intervals Co, B\, Ci, B%, . . . , 13%, C& based on the location of the braid clusters in w. We define the 
rth interval C r to be [p, q] where p is the rightmost column of the rth braid cluster and q is the leftmost 
column of the (r + l)st braid cluster, setting p = 1 for r = and q = n for r = k. The Bi then consist 
of the internal columns of the ith braid cluster. In particular, if w is freely braided then every Bi = 0. If 
w is fully commutative, then Co = {1, ... , n}. 

For example, suppose w is given by the contracted reduced expression 

{s5){siS-2S2,S i S2,S2Sl)(s & S 5 S 9 )(s 1 SsS 1 )(sQ). 

Then the heap of w is drawn below, together with its column partition. The braid clusters are shown in 
grey. 




C ] [Si ] [Ci ] [C 2 

Observe that the entries appearing in columns Bi are completely determined by the positions of the 
entries from the rightmost column in Cj_i and the leftmost column in Ci, as these are the ends of the 
braid cluster supported by Bi. 

Next, we define a map it : {Co, C\, . . . , C&} — > S n+ i to project each interval of columns to a permu- 
tation. If k = 0, then 7r(Co) = Co- Otherwise, observe that the leftmost column in each C for i > is 
the rightmost column of some braid cluster, so it consists of a single entry. The rightmost column q in 
each Ci fori<k is the leftmost column of some braid cluster, so it consists of two entries, and there 
can be no entries between them in column q — 1 by Definition 1.4. Hence, we may define a permutation 
7r(C), whose heap is obtained from the heap of w restricted to the columns of C by collapsing the two 
entries in column q to a single entry. 
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In the example above, vr(Ci) has the heap 



Next, we claim that for each i, 
(2.1) 7r(Q) G 



[Ci i 

r^ Co i iffc=o, 

L\c \ if k > and i = 0, 
M| C .| if fc > and 1 < i < fc, 
if k > and i = fc. 



In particular, we show that each ir(Ci) corresponds to a fully commutative permutation that heap-avoids 
the patterns from H. 

The claim is clear if k = 0, so suppose k > 0, and let Cj = [p, g] with < i < k. Then 7r(Cj) is fully 
commutative because every minimal pair of entries in 7r(Q) has a distinct resolution by property (2) in 
Definition 1.8 from Proposition 1.9. 

Next, suppose there exists an instance of a heap-pattern h € H among the entries of 7r(Cj). Then we 
must have such a heap-instance in w which is a contradiction. This follows because the heap of w with 
respect to the commutativity class where the (i + l)st braid cluster is of the form 

Sm+k+lSm+k ■ ■ ■ ■Sm+lSmSm+1 • ■ ■ •Sm+fc^m+fc+l 

has precisely the entries of 7r(Cj) in columns Q, and so w heap-contains h. As this cannot occur, each 
7r(Cj) heap-avoids the patterns from H. 

It follows from the construction that if k > then it (Co) has an entry in its rightmost column, 7r(Cfc) 
has an entry in its leftmost column, and every other 7r(Cj) has entries in both the leftmost and rightmost 
columns. Otherwise, k = 0, and tt(Cq) = Cq € F\C \- Thus, we have proved (2.1). 




Figure 3. Internal columns of a braid cluster 

Conversely, suppose there exists a fixed a column partition {Co, B±, Ci, . . . , C^} as in the begin- 
ning of the proof and we are given either a single heap G F\C \ if & = or a sequence of heaps 

(2.2) c^M l \c^M 2 \...M k \c {k) 

where k > 0, c(°) € ^|C |> c ^' ) e ^|C fc |> every other cW g Mi^.i and each 6W is a heap fragment on 
columns with the canonical form shown in Figure 3. Then, we may form a maximally clustered 
permutation w with k braid clusters as follows. If k = 0, take w to be the permutation whose heap is 
. Otherwise, for each i < k we expand the single entry in the rightmost column of to a pair of 
entries. This has the effect of reversing the ir map on each cW , which we denote by tt^ 1 (cW ). Then, glue 
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the columns together in order to form a single heap, so that the rightmost pair of entries in each ir 1 (cW ) 
surrounds the leftmost pair of entries in if any, or the unique leftmost entry in it' 
is empty Also, each &W is glued to vr-^c^) so that the pair of entries in the rightmost column of U- 1 ' 
suiTounds the unique entry in the leftmost column of tt~ 1 (c^). We call the permutation to which this 
heap corresponds w. 

Every minimal pair of entries x and y from an internal column of 7T -1 (cW) must have a distinct 
resolution because is fully commutative. Hence, we observe that the heap we constructed is reduced 
and maximally clustered because it has the form given in Proposition 1.9. 

Next, observe that if w heap-contains a pattern h G H then the pattern instance must lie in one of the 
cW To see this, let x and y be a minimal pair of entries lying in column i S [2, n — 1] of h. Because 
h is fully commutative, x and y have a distinct resolution, so there exist entries from columns i + 1 
and i — 1 lying strictly between x and y in the heap poset. However, no column in the heap of any 
commutativity class for a braid cluster has this property by the uniqueness statement in Definition 1.4, 
since each commutativity class for a braid cluster has a reduced expression representative that is itself 
a braid cluster. Thus, no heap-instance of h uses the internal columns of a braid cluster. Since we are 
assuming that each cW heap-avoids the patterns from H, we have that w heap-avoids the patterns from 
H. 

Finally, note that if we compose the constructions we have given above in either order, then we obtain 
the identity. Hence, we have shown a bijection and the enumerative formulas follow. F(x) contributes 
terms of the form where k = 0. Using Lemma 2.3, we obtain the generating function that counts 
sequences of the form given in (2.2) as the product 

L(x) ■ • — 1 • L(x) 

V ' l-x 1 - M(x) ■ K ; 

because each term of the sequence in (2.2) is independent. Multiplying the numerator and denominator 
of the third factor by (1 — x), we obtain the second result. In the case where w is freely braided, each 
&W term is 0, so the corresponding generating function is simply 

L{X) ■ l^M{x) ■ L{X) 

which gives the first result. □ 

Corollary 2.5. Let H be a finite set of connected fully commutative permutations each of whose heaps 

contains at least two entries in each internal column. If F(x) = Xm>o l^n+l C^OI iS a ran onal 
(respectively, algebraic) generating function, then 

£ |5 {[3421],[4231],[4312],[4321]} (jff)| %n ^ £ j ^[3421], [4312], [4321] } j ^ 
n>0 n>0 

are also rational [respectively, algebraic). 



Using Theorems 2.2 and 2.4, we can enumerate several interesting classes characterized by permuta- 
tion pattern avoidance. Note that we index the coefficients in the generating functions by the rank of the 
Coxeter group rather than by the number of entries appearing in the 1-line notation. 
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Generating function 


Initial sequence 


[321] -avoiding 


l-2x— \A~4a: 
2x 2 


1 + 2x + 5x 2 + 14k 3 + 42k 4 + 132k 5 + 429k 6 + 1430k 7 + . . . 


Tj(t\ for 
H = % 


l-3x-(x-l)^l-Ax 
2x 2 


T -4- 3t 2 -1- Qk 3 4- 98t- 4 -t- QOt 5 -1- 9Q7t" 6 A- 1 001 r 7 4- 


]\/f ( ry> \ f nr 

H = 


l-Ax+3x 2 -2x 6 -(x-iy s/l-Ax 
2x 2 


ZK 4- DK -+- lyK + DZK + ZUYK 4- 704k + . . . 


Freely-braided 


2x-2x 2 -2x^l-Ax 
-l+Ax-x 2 +2x i +(x- X]' z Jl-Ax 


1 + 2k + 6k 2 + 20k 3 + 71k 4 + 260k 5 + 971k 6 + 3674k 7 + . . . 


Maximally- 
clustered 


2x 

-1+Ax~2x 2 +^1-Ax 


1 + 2k + 6k 2 + 21k 3 + 78k 4 + 298k 5 + 1157k 6 + 4539k 7 + . . . 


[321] -hexagon 
avoiding 


-x b +x' 1 +3x A -Ax 2 +Ax-l 
x b -Ax b -Ax i +9x' 6 ~llx 2 +6x-l 


1 + 2k + 5k 2 + 14k 3 + 42k 4 + 132k 5 + 429k 6 + 1426k 7 + . . . 


L(x) for if = 
{[46718235]} 


AX -\-£X — AX +OX — X 

x b -Ax b -Ax i +9x i ~llx 2 +&x-l 


k + 3k 2 + 9k 3 + 28k 4 + 90k 5 + 297k 6 + 997k 7 + . . . 


M(x) for if = 
{[46718235]} 


-x' +2x° +Ax b -bx*+Gx- s -2x z 
x b -Ax b -Ax*+9x'- s -llx 2 +6x-l 


2k 2 + 6k 3 + 19k 4 + 62k 5 + 207k 6 + 700k 7 + . . . 


Freely-braided 
hexagon- 
avoiding 


n — 7z — r — ^ — n t ~ — t : 

-x b ~2x°+2x i +x- i -3x 2 +Ax-l 

x 7 -x a -8x 5 +x 4 +3x 3 -9x' 2 +6x-l 


1 + 2k + 6k 2 + 20k 3 + 71k 4 + 260k 5 + 971k 6 + 3670k 7 + . . . 


Maximally- 
clustered 
hexagon- 
avoiding 


3x b +x 4 -5x J +7x 2 -5x+l 
-3x b +Ax b +8x' t -lAx :i +15x 2 -7x+l 


1 + 2k + 6k 2 + 21k 3 + 78k 4 + 298k 5 + 1157k 6 + 4535k 7 + . . . 



Remark 2.6. The first line is the Catalan generating function which appears in this context by [SS85], 
while the sixth line is Theorem 1.1 due to [SW04]. The freely braided permutations have previously 
been enumerated in [Man04], while the other formulas seem to be new. They follow from Theorem 2.2 
together with Lemma 2.3 and Theorem 2.4 by taking H = and H = {[46718235]}, respectively. 

Corollary 2.7. The number b n of freely-braided hexagon-avoiding permutations in S n satisfies the re- 
currence 

b n+ i = 6b n - 96 n _i + 36 n _ 2 + K-3 ~ 86 n -4 - K-s + b n -6 
and the number m n of maximally-clustered hexagon-avoiding permutations in S n satisfies the recurrence 

m n+ i = 7m n - 15m n _i + 14m n _ 2 - 8m„_3 - 4m n _ 4 + 3m n _ 5 

for all n > 9, with initial conditions given by Figure 1. 

3. Diamond reductions 

In this section, we restrict to considering fully commutative permutations. 

Definition 3.1. Suppose that h is a connected, fully commutative permutation whose heap contains at 
least two entries in each internal column. We say that every minimal pair of entries together with the 
entries of their distinct resolution forms a minimal diamond inside the heap of h. Form a new heap whose 
entries correspond to minimal diamonds. The poset structure of the new heap is inherited from the poset 
structure on the minimal diamonds in the heap of h by taking the transitive closure of the relation that 
two minimal diamonds are related if they share an edge. The heap obtained in this way from h is called 
the diamond reduction of h. 
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Pictorially, the heap of the diamond reduction of h is obtained from the heap of h by adding entries at 
the centers of all minimal diamonds and then erasing the heap of h. 

Example 3.2. The diamond reduction of the hexagon S5S6S7S3S4S5S6S2S3S4S5S1S2.S3 is the 3-hexagon 
•S4.s5s2.s3s4.s1.s2- 

• • 

/ \ / \ 

• » • • • 

/ \ / \ / \ 
• •••••• 

\ / \ / \ / 

• » • • • 
\ / \ / 

• • 

Proposition 3.3. The diamond reduction is a bijection from the set of connected fully commutative heaps 
on n columns with at least two entries in each internal column to the set of connected fully commutative 
heaps on n — 2 columns. 

Proof. Let h be a connected fully commutative heap on n columns with at least two entries in each 
internal column. The diamond reduction g of h gives a reduced heap because we can represent the 
minimal diamonds by their maximal entries (rather than the centers) and the result is a convex subposet 
of the heap of h. The heap of g is also connected since every internal column of the heap of h has a 
minimal diamond. The diamond reduction can be reversed by forming a heap with minimal diamonds 
centered at the heap entries of g. Hence, the diamond reduction of h is fully commutative because any 
short-braid instance in the heap of g would imply the existence of a short-braid in h when we consider 
the minimal diamonds centered at the entries of the heap of g. We also reduce the number of columns by 
2 since only the internal columns of the heap of h support minimal diamonds that become entries in the 
heap of g. □ 

Our main goal in this section is to describe how the generating functions for { | {h)\} and { | sjf+i (g) \ } 
are related when g is the diamond reduction of h. 

Lemma 3.4. Let H be a set of connected, fully commutative permutations. Suppose 

F(x) = \S [ n+}(H)\ x n and F c {x) = \{p G sjjji C*0 : P is connected }| x n , 

n>0 n>0 

with F(0) = 1 and F c (0) = 0. Then the generating functions are rationally related according to 

. . 1 + FJx) , . F(x) - xF(x) - 1 

Fix) = cK J, . and FJx) = ——^ • 

v ' l-x-xF c (x) y ' l + xF(x) 

Proof. To see this, observe that every not-necessarily-connected ^-avoiding heap on n columns decom- 
poses uniquely into connected components, each of which is counted by F c (x). Conversely, if we have 
an ordered collection C of connected permutations that all heap-avoid the patterns from H, then we can 
place them together from left to right, to obtain a heap which has the heaps from C as its connected 
components. 

The generating function identity that we obtain from this observation is 

1 1 1 1 

F W = — + — ~ x • F ^ ■ X -^F c {x)\-x 

Specifically, we either have some connected components, or none. The initial term corresponds to 
the case where we have none. Otherwise, we have zero or more initial empty columns, followed by at 
least one connected component, followed by zero or more copies of empty columns together with other 
connected components, finally followed by zero or more empty columns on the right side of the heap. 
This is the second term. This formula simplifies to the one given in the statement and can be inverted. □ 
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One fundamental subclass of the fully commutative permutations are those with no minimal diamonds 
at all. It is straightforward to see that this conesponds to having at most one entry in each column, or 
equivalently to heap-avoiding S2S1S3S2. This pattern class was previously enumerated in [Fan98] and 
[Wes96]. Also, [Ten07] enumerates these permutations using the statistic of Coxeter length. 

Lemma 3.5. We have 

x — *> 1 ~r35ii.rad.i9i . « x-^ , -.rami , „ 



K [321],[3412], „_V^ K [321], » 



/^^n+l I - WWVI ~ 1 _ 3x + x 2- 

n>0 n>0 

[3211 

Proof. The second equality follows from Theorem 2.2. Let w be a connected element of (S2S1S3S2). 
Then the heap of w is a lattice path consisting of n — 1 steps that are either "up" or "down." As each step 
is independent, the generating function for these is x \ x - Applying Lemma 3.4 gives the result. □ 

We are now in a position to state and prove our main result in this section. 

Theorem 3.6. Suppose that h is a connected, fully commutative permutation whose heap contains at 
least two entries in each internal column and let g be the diamond reduction of h. If 

G{x) = xH > G ^ x ) = E \iP € sl nS(9) ■■ P » connected }\x n , 

n>0 n>0 
rooil 

and F(x) = Xm>o l^n+i (^)l x11 ' with G(0) = 1 = ^(0) andG c (0) = 0, then the generating functions 
are rationally related according to 

1 — x — xG c (x) 1 



(3.1) F(x) 



1 - 3x + x 2 + (x 2 - x)G c {x) l-2x- x 2 G(x) 



Proof. Suppose w £ S^l 1 (h). We assign each of the columns {1, ... n} of the heap of w to intervals 
Eq, Di, Ei, . . . , Dk,Ek according to whether the column supports more than one entry. The intervals 
Di are defined to be precisely those that support maximal connected fully commutative subheaps of the 
heap of w such that the internal columns of Dj each support at least two entries. The remaining intervals 
Ei consist of columns that support at most one entry. In particular, the Ei may include empty columns. 
For example, suppose 

W = SlS2Sl2S8S9Sl0SllS 5 S 6 S7S8S4S5S6- 

Then the heap of w is drawn below, together with its column assignments. 



[ E ] [Ei] [ E 2 

[ Di ][ D 2 ] 



Let w\Di denote the restriction of the heap of w to the columns in the interval Di. By the hypotheses 
given, each w\D t must heap-avoid h. Moreover, we can form the diamond reduction of w\Di whose 
heap must be connected, heap-avoids g, and has 2 fewer columns than w\oi- Each u?^ must heap-avoid 
S2S1S3S2 and contains entries in the extremal columns of Ei that are shared with Di or Di + \. 
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Conversely, suppose there exists a fixed column assignment Eq, D\, E\, . . . , D^, and we are given 
a sequence of heaps e , d±, e±, . . . , dfc, e& such that eo G Swu. i( s 2SiS3S2) with one entry in the right- 
most column of Eq, di G ^ij>[-i(#) an< ^ ^ * s connecl:e d for 1 < z < fc, e.; G 5 , |^?| L ^ 1 (s2'5iS3S2) for 
1 < i < k — 1 with one entry in each of the extremal columns of Ei and € S^ l ^ +l {s2Sis^S2) with 
one entry in the leftmost column of Ej.. Then we can apply Proposition 3.3 to form the reverse diamond 
reduction di of di and glue these heaps together to obtain an element of S^^(h). Specifically, we iden- 
tify the unique entry in the rightmost column of with the unique entry in the leftmost column of di + ± 
for each < i < k — 1, and we also identify the unique entry in the leftmost column of with the 
unique entry in the rightmost column of di for each 1 < % < k. 

If we compose the constructions we have given in either order, we obtain the identity. Hence, we have 
shown a bijection. 

Let E(x) be the generating function appearing in Lemma 3.5. Define Elr(x) = E{x) — xE(x) — 1 
and Em(x) = E(x) — 2xE(x) + x 2 E(x) — 1 + x. It is straightforward to verify using inclusion- 
exclusion that Elr(x) counts the number of S2SiS3S2-avoiding heaps with one entry in at least one of 
the two extremal columns, and Em{%) counts the number of S2SiS3S2-avoiding heaps with one entry in 
both of the extremal columns. 

Putting all of these observations together, we have the generating function identity 

(3.2) F(x) = E(x) + E LR (x)- 1 G c (x)E LR (x). 

1 - G c (x)E M {x) 

Here, E{x) contributes terms of the form where k = 0. This formula simplifies to the first equality given 
in the statement. The second equality follows from Lemma 3.4. □ 

Remark 3.7. In the nicest cases, this result allows us to reduce the problem of enumerating permutations 
that avoid a given heap pattern to counting lattice paths that avoid a certain consecutive subpath. This 
problem is well known to have a rational generating function and the transfer matrix method can be used 
to find the generating function explicitly. See [Sta97, Example 4.7.5] for details. 

Example 3.8. 

V I c[321], [356124], [456123] , „ _ I S [32l] ( Sa ScSoSvS, Si SoS*) I X n ~ 1 - — + — - — 

n>0 n>0 

The first equality follows from Theorem 2.2. The heap pattern p = S4S5S2S3S4S1S2S3 has a diamond 
reduction to q = siS3S2- 

• • • 

/ \ / \ 
• • • • • 
\ / \ / 

• • 

The number of connected permutations that heap-avoid q is the same as the number of lattice paths that 
never contain a consecutive up-down subpath. It is straightforward to see that there are n such lattice 
paths that use n nodes. The corresponding generating function is G c {x) = ^ x yi and substituting this 
into Equation (3.1) yields the result. 



Example 3.9. 

E, [321], [46718235], [46781235], [56718234], [56781234] , „ , [321] / 

n>0 n>0 
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1 — Ax + Ax 2 - 3x 3 - x 4 + x 5 
~~ 1 - 6x + llx 2 - 9x 3 + Ax 4 + Ax 5 - x 6 ' 

recovering Theorem 1.1 

The first equality follows from Theorem 2.2. The hexagon pattern has a diamond reduction to 
q = S4S5S2S3S4S1S2 as shown in Example 3.2. However, the diamond reduction of q is S1S3 which 
is disconnected. Hence, we must enumerate the permutations heap-avoiding q directly. We use the same 
column assignment as in the proof of Theorem 3.6. In order to heap-avoid q, the intervals of columns 
supporting diamonds must restrict to fully commutative permutations whose diamond reduction is a con- 
nected monotonic lattice path. Also, we can no longer glue two diamond-containing regions together 
along a trivial lattice path with one column. Hence, we must modify the Em{x) term of Equation (3.2) 
by subtracting x. 

The monotonic lattice paths are counted by G c (x) = — x. Substituting this into Equation (3.2) 
along with Em{x) = E(x) — 2xE(x) +x 2 E(x) — 1 gives the generating function for all g-heap-avoiding 
permutations. Applying Theorem 3.6 then yields the result. 
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